Situation determining apparatus, situation determining  method, situation determining program, abnormality determining apparatus, abnormality determining method, abnormality determining program, and congestion estimating apparatus

ABSTRACT

A congestion estimating apparatus includes an area dividing unit that divides a moving image into partial areas. A movement information determining unit determines whether there is movement, and a person information determining unit determines whether there is a person, in each partial area. A staying determining unit determines a state for each partial area. The staying determining unit determines the state as a movement area in which there is a movement of person when there is movement and there is a person; and determines the state as a noise area when there is movement and there is no person; and determines the state as a staying area in which there is a person who is staying when there is no movement and there is a person; and determines the state as a background area in which there is no person when there is no movement and there no person.

This application is a division of U.S. patent application Ser. No.12/601,950 filed Nov. 25, 2009, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present invention relates to a situation determining apparatus, asituation determining method, a situation determining program, anabnormality determining apparatus, an abnormality determining method,and an abnormality determining program capable of analyzing an imagecaptured in a public space in which a plurality of persons is moved,such as a station or an airport, to detect the degree of congestion orthe movement situation of the persons.

In addition, the present invention relates to a congestion estimatingapparatus that estimates the degree of congestion of persons on thebasis of an image, and more particularly, to a congestion estimatingapparatus that determines the kind of the staying state or movementstate of persons to detect an abnormal state.

BACKGROUND ART

In recent years, with an increasing demand for safety and security,monitoring cameras have been installed in a public space such as astation or an airport, and important facilities. In the related art, anobserver monitors a monitoring camera all the time. Therefore, there isan attempt to reduce efforts for monitoring and improve monitoringefficiency using image recognition in order to prevent the observer fromoverlooking the monitoring cameras due to an increase in the number ofmonitoring cameras and the fatigue of the observer.

Patent Document 1 and Patent Document 2 disclose a technique forcounting the number of persons in a monitoring place. Patent Document 1discloses a technique that extracts the background using a backgrounddifference and counts the number of persons who move across a monitoringarea which is orthogonal to a person's passage. A plurality ofmonitoring areas is prepared, and a variation in count value between themonitoring areas is used to accurately count the persons even thoughdisturbance occurs.

Patent Document 2 discloses a technique in which a camera is providedabove a passage such that the optical axis is aligned with the verticaldirection, a motion vector is extracted at a boundary line provided onthe image in order to count the number of persons, and a verticalcomponent of the motion vector with respect to the boundary line isintegrated to count the number of persons passing through the passage.

Patent Document 3 discloses a technique that extracts featurescorresponding to the number of persons from an image without countingthe number of persons to calculate the degree of congestion. In PatentDocument 3, the number of changes in each pixel or each local area for apredetermined amount of time is calculated and the degree of congestionis calculated on the basis of the number of changes on the assumptionthat, “when there are a large number of passengers, the number ofchanges between images captured at different times is increased”.

In addition, in the related art, various techniques have been proposedwhich estimate the degree of congestion of persons in an image. Forexample, a technique has been proposed which calculates motion vectors,calculates the integral value of the motion vectors, and counts thenumber of persons in unit of the integral value (for example, see PatentDocument 4).

A technique has been proposed which detects the heads of the persons,measures the number of heads, and estimates the degree of congestion(for example, see Patent Document 2).

A technique has been proposed which estimates the degree of congestionusing the area of the background extracted by an inter-frame differenceor background difference process (for example, see Patent Document 3 orPatent Document 5).

-   Patent Document 1: JP-A-2002-074371-   Patent Document 2: JP-A-2005-135339-   Patent Document 3: JP-A-2004-102380-   Patent Document 4: JP-A-2005-128619-   Patent Document 5: JP-A-11-282999

DISCLOSURE OF THE INVENTION Problem that the Invention is to Solve

However, the above-mentioned techniques according to the related arthave the following problems. That is, the technique disclosed in PatentDocument 1 uses the background difference. Therefore, it is difficult toapply the technique to a place in which there is a large variation inillumination, and it is difficult to count the persons one by one duringcongestion. In the technique disclosed in Patent Document 2, similarly,it is difficult to count the persons one by one during congestion. InPatent Document 3, since it is premised that persons move, it isdifficult to calculate the degree of congestion in a situation in whichthere are moving persons and standing persons.

The method according to the related art which integrates the motionvectors and measures the number of persons in unit of the integral valueis a method of cutting out each person. In the method, when there are asmall number of persons, it is possible to relatively accuratelyestimate the degree of congestion. However, in a congestion situation,the persons overlap each other. Therefore, it is difficult to apply thistechnique to a congestion situation, and the accuracy of the estimationis lowered. When the motion vectors are extracted, a motion vector bynormal movement and a fine motion vector generated by, for example,noise have different sizes due to the angle of view of the camera.Therefore, it is necessary to set the threshold value of the size of themotion vector in advance.

The method of detecting the heads of the persons has a problem in that,when there are a small number of persons, it is difficult to relativelyaccurately estimate the degree of congestion, and when there are a largenumber of persons, the detection accuracy of the heads is lowered.

In the method of estimating the degree of congestion using the area ofthe background extracted by the inter-frame difference process, when thepersons remain stationary, the background is not extracted. In themethod of estimating the degree of congestion using the area of thebackground extracted by the background difference process, when thereare persons in most of the screen, it is difficult to accurately abackground area. The method is easily affected by the shaking of thecamera. In addition, in the method, a method of calculating the index ofa congestion situation (a staying area, a movement area, a normal area,a staying start state, a staying removal state, and a normal state) isnot disclosed, and it is difficult to check a partial congestionsituation.

The invention has been made in order to solve the above-mentionedproblems, and an object of the invention is to provide a situationdetermining apparatus, a situation determining method, a situationdetermining program, an abnormality determining apparatus, anabnormality determining method, and an abnormality determining programcapable of easily determining the situation of a monitoring place andthe degree of congestion.

The invention has been made in order to solve the above-mentionedproblems, and an object of the invention is to provide to a congestionestimating apparatus capable of easily and accurately estimating thedegree of congestion of persons on the basis of an image.

Means for Solving the Problem

In order to achieve the object, the present invention provides asituation determining apparatus for analyzing captured moving images ora plurality of captured still images to determine a movement situationand/or a degree of congestion of persons, the situation determiningapparatus including: a local image change ratio calculating unit thatcalculates a time change ratio of a brightness value in a local area ofthe captured images; and a situation determining unit that analyzes ahistogram of the time change ratios of a plurality of local areascalculated by the local image change ratio calculating unit anddetermines the movement situation of the persons and/or the degree ofcongestion of the persons.

According to the above-mentioned structure, the local change ratios of aplurality of local areas in the captured images are calculated, ahistogram of the local change ratios of the plurality of areas iscalculated, and the histogram is analyzed. Therefore, it is possible todetect the spatial characteristics (for example, the deflection ofmovement to one side) of a change ratio corresponding to the incidenceof the movement of an object, and it is possible to comprehensivelydetermine the situation of a monitoring place and the degree ofcongestion.

The present invention provides a situation determining apparatus foranalyzing captured moving images or a plurality of captured still imagesto determine a movement situation and/or a degree of congestion ofpersons, the situation determining apparatus including: an image inputunit that inputs moving images or a plurality of still images of personscaptured in an imaging target place; an image accumulating unit thataccumulates the images input by the image input unit; a local imagechange detecting unit that selects two images captured at a first timeinterval from the images accumulated in the image accumulating unit anddetects a change between the two images in each local area using areadivision information indicating a division method of dividing the imageinto a plurality of local areas; a local image change informationaccumulating unit that accumulates the change between the two imagesdetected by the local image change detecting unit as image changeinformation; a local image change ratio calculating unit that counts thenumber of changes between images given at a second time interval in eachlocal area on the basis of the image change information accumulated inthe local image change information accumulating unit, and calculates animage change ratio of each local area; a local image change ratioaccumulating unit that accumulates the image change ratios of aplurality of local areas calculated by the local image change ratiocalculating unit; a local image change ratio histogram calculating unitthat calculates a histogram of the image change ratios of the pluralityof local areas accumulated by the local image change ratio accumulatingunit; and a situation determining unit that analyzes the histogramcalculated by the local image change ratio histogram calculating unit todetermine the movement situation and or the degree of congestion of thepersons in the imaging target place.

According to the above-mentioned structure, the local change ratios of aplurality of local areas in the captured images are calculated, ahistogram of the local change ratios of the plurality of areas iscalculated, and the histogram is analyzed. Therefore, it is possible todetect the spatial characteristics (for example, the deflection ofmovement to one side) of a change ratio corresponding to the incidenceof the movement of an object, and it is possible to comprehensivelydetermine the situation of a monitoring place and the degree ofcongestion.

In the above-mentioned structure, the situation determining unitincludes a reference histogram storage unit and a histogram comparingunit.

In the above-mentioned structure, the situation determining unitincludes a feature extracting unit, an identification reference storageunit, and an identifying unit.

In the above-mentioned structure, the movement situation includes atleast a situation in which a moving route of the persons is deflected toone side.

According to the above-mentioned structures, it is possible to detectthe deflection of a moving route to one side using a simple process.Therefore, it is possible to estimate whether there is a line of personswaiting for the train and the congestion level of the persons in a placewhere there is a line of persons for waiting for the train, such as astation. Obviously, in a situation in which the moving route is notdeflected to one side, it is also possible to estimate the situation anda congestion level. When the deflection of the moving route to one sideis detected in a passage where persons are not generally lined up, it ispossible to estimate an obstacle to the free movement of persons.

The present invention provides an abnormality determining apparatus foranalyzing moving images or a plurality of still images captured by animaging unit that is provided at a platform of a station to determine anabnormal situation, the abnormality determining apparatus including: thesituation determining apparatus; a train arrival detecting unit thatdetects an arrival of a train to the platform; and an abnormalitydetermining unit which determines that abnormality occurs when it isdetermined that the moving route of the persons is deflected to the oneside as the situation determination result after a predetermined amountof time has elapsed from the acquisition of train arrival information bythe train arrival detecting unit, on the basis of a situationdetermination result of the situation determining apparatus and thetrain arrival information obtained by the train arrival detecting unit.

According to the above-mentioned structure, it is possible to determinean abnormal congestion state different from a normal state on the basisof the kind of condition or the degree of congestion obtained by thesituation determining apparatus.

In the above-mentioned structure, the abnormality determining apparatusfurther includes a notifying unit that gives the determination result ofthe abnormality determining unit to a predetermined contact address whenthe abnormality determining unit determines that abnormality occurs.

According to the above-mentioned structure, it is possible to provideauxiliary information to the observer or rapidly transmit it to apredetermined contact address.

The present invention provides a situation determining method ofanalyzing captured moving images or a plurality of captured still imagesto determine a movement situation and/or a degree of congestion ofpersons, the situation determining method including: a local imagechange ratio calculating step of calculating a time change ratio of abrightness value in a local area of the captured images; and a situationdetermining step of analyzing a histogram of the time change ratios of aplurality of local areas calculated by the local image change ratiocalculating unit and determining the movement situation of the personsand/or the degree of congestion of the persons.

According to the above-mentioned method, the local change ratios of aplurality of local areas in the captured images are calculated, ahistogram of the local change ratios of the plurality of areas iscalculated, and the histogram is analyzed. Therefore, it is possible todetect the spatial characteristics (for example, the deflection ofmovement to one side) of a change ratio corresponding to the incidenceof the movement of an object, and it is possible to comprehensivelydetermine the situation of a monitoring place and the degree ofcongestion.

In the situation determining method, the movement situation includes atleast a situation in which a moving route of the persons is deflected toone side.

According to the above-mentioned method, it is possible to detect thedeflection of a moving route to one side using a simple process.Therefore, it is possible to estimate whether there is a line of personswaiting for the train and the congestion level of the persons in a placewhere there is a line of persons for waiting for the train, such as astation. Obviously, in a situation in which the moving route is notdeflected to one side, it is also possible to estimate the situation anda congestion level. When the deflection of the moving route to one sideis detected in a passage where persons are not generally lined up, it ispossible to estimate an obstacle to the free movement of persons.

The present invention provides an abnormality determining method ofanalyzing moving images or a plurality of still images captured by animaging unit that is provided at a platform of a station to determine anabnormal situation, the abnormality determining method including: asituation determining step of performing the situation determiningmethod; a train arrival detecting step of detecting an arrival of atrain to the platform; and an abnormality determining step ofdetermining that abnormality occurs when it is determined that themoving route of the persons is deflected to the one side as thesituation determination result after a predetermined amount of time haselapsed from the acquisition of train arrival information by the trainarrival detecting unit, on the basis of a situation determination resultof the situation determining apparatus and the train arrival informationobtained by the train arrival detecting unit.

According to the above-mentioned method, it is possible to determine anabnormal congestion state different from a normal state on the basis ofthe kind of condition or the degree of congestion obtained by thesituation determining apparatus.

The present invention provides a situation determining program thatanalyzes captured moving images or a plurality of captured still imagesto determine a movement situation and/or a degree of congestion ofpersons, the situation determining program allowing a computer toexecute: a local image change ratio calculating step of calculating atime change ratio of a brightness value in a local area of the capturedimages; and a situation determining step of analyzing a histogram of thetime change ratios of a plurality of local areas calculated by the localimage change ratio calculating unit and determining the movementsituation of the persons and/or the degree of congestion of the persons.

According to the above-mentioned program, the local change ratios of aplurality of local areas in the captured images are calculated, ahistogram of the local change ratios of the plurality of areas iscalculated, and the histogram is analyzed. Therefore, it is possible todetect the spatial characteristics (for example, the deflection ofmovement) of a change ratio corresponding to the incidence of themovement of an object, and it is possible to comprehensively determinethe situation of a monitoring place and the degree of congestion.

In the situation determining program, the movement situation includes atleast a situation in which a moving route of the persons is deflected toone side.

According to the above-mentioned program, it is possible to detect thedeflection of a moving route to one side using a simple process.Therefore, it is possible to estimate whether there is a line of personswaiting for the train and the congestion level of the persons in a placewhere there is a line of persons for waiting for the train, such as astation. Obviously, in a situation in which the moving route is notdeflected to one side, it is also possible to estimate the situation anda congestion level. When the deflection of the moving route to one sideis detected in a passage where persons are not generally lined up, it ispossible to estimate an obstacle to the free movement of persons.

An abnormality determining program that analyzes moving images or aplurality of still images captured by an imaging unit that is providedat a platform of a station to determine an abnormal situation, theabnormality determining program allowing a computer to execute: asituation determining step of performing the situation determiningmethod; a train arrival detecting step of detecting an arrival of atrain to the platform; and an abnormality determining step ofdetermining that abnormality occurs when it is determined that themoving route of the persons is deflected to the one side as thesituation determination result after a predetermined amount of time haselapsed from the acquisition of train arrival information by the trainarrival detecting unit, on the basis of a situation determination resultof the situation determining apparatus and the train arrival informationobtained by the train arrival detecting unit.

According to the above-mentioned program, it is possible to determine anabnormal congestion state different from a normal state on the basis ofthe kind of condition or the degree of congestion obtained by thesituation determining apparatus.

The present invention provides a congestion estimating apparatusincluding: an image generating unit that converts an image of variousscenes or an image captured by a camera into a digital image and outputsthe digital image; an area dividing unit that divides an input imageinto partial areas; a movement information generating unit thatgenerates movement information from the image output from the imagegenerating unit; a texture information generating unit that generatestexture information of the image output from the image generating unit;a reference movement information generating unit that stores and updatesreference movement information, which is a reference for movement ineach partial area; a reference texture information generating unit thatstores and updates reference texture information for determining whetherthere is a person in each partial area; a storage unit that stores thereference movement information and the reference texture information; amovement information determining unit that compares the movementinformation output from the movement information generating unit withthe reference movement information generated by the reference movementinformation generating unit to determine whether there is a movement ineach partial area; a texture information determining unit that comparesthe texture information output from the texture information generatingunit with the reference texture information generated by the referencetexture information generating unit to determine whether there is thesame texture information as a person in each partial area; and a stayingdetermining unit that receives determination results from the movementinformation determining unit and the texture information determiningunit to determine whether there is a person in each area.

According to the above-mentioned structure, the movement informationdetermining unit determines whether there is a movement in each area.Even when there is no movement, the texture information determining unitcan determine whether there is a person from texture similarity.Therefore, it is possible to estimate the state of each area, such as amovement area, a staying area, a stationary area, and an area in whichthere is no person. Then, the staying determining unit can determine thedegree of congestion on the basis of each information item.

In the above-mentioned structure, the congestion estimating apparatusfurther includes a timing generating unit that receives the imagegenerated by the image generating unit and determines whether there is aperson from movement information, wherein, only when it is determinedthat there is a person, the timing generating unit gives update timingto the reference movement information generating unit and to thereference texture information generating unit.

According to the above-mentioned structure, the reference movementinformation of the reference movement information generating unit andthe reference texture information of the reference texture informationgenerating unit are updated at each update timing. Therefore, even whenan environment is changed, it is possible to determine whether there isa movement or texture according to the variation in the environment. Asa result, it is possible to accurately perform determination all thetime.

In the above-mentioned structure, the timing generating unit detectsapproach timing of a vehicle, and gives the update timing to thereference movement information generating unit and to the referencetexture information generating unit at each approach timing.

According to the above-mentioned structure, the reference movementinformation of the reference movement information generating unit andthe reference texture information of the reference texture informationgenerating unit are updated at each vehicle approach timing. Therefore,it is possible to determine whether there is a movement or texture onthe basis of the movement of persons or the texture of the personsbefore and after the vehicle approaches.

In the above-mentioned structure, the reference movement informationgenerating unit samples reference movement information at the timingnotified by the timing generating unit to set a threshold value of thereference movement information, the movement information determiningunit determines that there is a movement when the reference movementinformation is more than a threshold value of the reference movementinformation; whereas the movement information determining unitdetermines that there is no movement when the reference movementinformation is not more than the threshold value of the referencemovement information.

According to the above-mentioned structure, the reference movementinformation of the reference movement information generating unit isupdated at each update timing, and the threshold value of the referencemovement information is set at the update timing. Therefore, it ispossible to determine the movement information on the basis of movementat the update timing.

In the above-mentioned structure, the texture information determiningunit performs a frequency conversion process on input information todetermine similarity in a frequency domain.

According to the above-mentioned structure, it is possible to evaluatesimilarity on the basis of the outline or silhouette of a person.

In the above-mentioned structure, the reference texture informationgenerating unit samples reference texture information at the timingnotified by the timing generating unit to set the reference textureinformation, and the texture information determining unit determinessimilarity between the texture information generated by the textureinformation generating unit and the reference texture informationgenerated by the reference texture information generating unit, and whenit is determined that the texture information is similar to thereference texture information, the texture information determining unitdetermines that there is a person.

According to the above-mentioned structure, the reference textureinformation of the reference texture information generating unit isupdated at each update timing, and it is possible to determine thesimilarity of texture on the basis of the texture at the update timing.

In the above-mentioned structure, the staying determining unit receivesthe determination result of the movement information determining unitand the determination result of the texture information determining unitto output a state of any one of a staying area, a movement area, a noisearea, and a background area as the state of each area.

According to the above-mentioned structure, it is possible to output astaying area, a movement area, a noise area, and a background area asthe state of each area. Therefore, when performing a congestionestimating process, the congestion determining unit can measure thedegree of congestion for each kind of area. In addition, it is performcalculation only by counting the overall degree of congestion.

In the above-mentioned structure, the congestion estimating apparatusfurther includes an abnormality determining unit that receivesinformation output from the staying determining unit, and analyzes eachinput state to determine whether abnormal congestion occurs.

According to the above-mentioned structure, it is possible to determinewhether the degree of congestion of the entire imaging environment isnormal or abnormal on the basis of the state of each area.

In the above-mentioned structure, the abnormality determining unitcounts various states of each area, the staying area, the movement area,the noise area, and the background area output from the stayingdetermining unit, and when a congestion index, which is the sum of thenumber of staying areas and the number of moving areas, is not reducedby a predetermined threshold value or more after the approach timing ofthe vehicle obtained by the timing generating unit, the abnormalitydetermining unit determines that abnormality occurs.

According to the above-mentioned structure, the state of each area iscounted after the approach timing of the vehicle to calculate thecongestion index. Therefore, it is possible to determine whether acongestion state is normal or abnormal.

In the above-mentioned structure, the abnormality determining unitcounts various states of each area, the staying area, the movement area,the noise area, and the background area output from the stayingdetermining unit, and when the ratio of the staying area is more than apredetermined value, the abnormality determining unit determines thatabnormality occurs.

According to the above-mentioned structure, the state of each area iscounted all the time to calculate the congestion index, therebycalculating the ratio of the staying area. Therefore, it is possible todetect abnormality on the basis of the ratio of the staying area.

In the above-mentioned structure, the abnormality determining unitcounts various states of each area, the staying area, the movement area,the noise area, and the background area output from the stayingdetermining unit, and the abnormality determining unit determines thetendency of the movement of the persons, such as a staying start,staying removal, and a normal state, from the ratios of the staying areaand the movement area in time series.

According to the above-mentioned structure, the state of each area and atime-series variation are counted all the time to calculate thecongestion index. Therefore, it is possible to determine the tendency ofthe movement of persons, such as a staying start, staying removal, and anormal state. In addition, it is possible to issue a warning beforeabnormal congestion occurs, and notify the start of the removal ofabnormal congestion.

Advantage of the Invention

According to the invention, the local change ratios of a plurality oflocal areas in the captured images are calculated, a histogram of thelocal change ratios of the plurality of areas is calculated, and thehistogram is analyzed. Therefore, it is possible to detect the spatialcharacteristics (for example, the deflection of movement to one side) ofa change ratio corresponding to the incidence of the movement of anobject, and it is possible to comprehensively determine the situation ofa monitoring place and the degree of congestion.

Even when the movement of persons is deflected to one side after apredetermined amount of time has elapsed from the arrival of the train,it is determined that abnormality occurs on the basis of both thedetermined comprehensive situation or the degree of congestion, and thearrival information of the information. It is possible to provideauxiliary information to the observer or rapidly transmit it to apredetermined contact address.

According to the invention, the reference movement amount of themovement information is automatically set. Therefore, it is possible todiscriminate a state in which there is a movement from a state in whichthere is no movement. In addition, it is possible to determine whetherthere is a movement and uses texture to determine similarity, therebydiscriminating various states of each area, a staying area, a movementarea, a noise area, and a background area. Further, it is possible touse the state of each area to estimate the degree of congestion, andprovide the indexes of congestion situations (a staying area, a movementarea, a normal area, a staying start state, a staying removal state, anda normal state) and information about an abnormal state. Therefore, itis possible to easily and accurately estimate the degree of congestionof the persons using an image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram schematically illustrating the structure of asituation determining apparatus according to a first embodiment of theinvention.

FIG. 2 is a diagram illustrating the installation of the situationdetermining apparatus according to the first embodiment of the inventionat a railroad station.

FIG. 3 is a diagram illustrating an image captured by a camera CMaccording to the first embodiment of the invention.

FIG. 4 is a flowchart illustrating a situation determining methodperformed by the situation determining apparatus according to the firstembodiment of the invention.

FIG. 5 is a diagram illustrating the time-series relationship betweenframe images accumulated in an image accumulating unit and local changeinformation accumulated in a local image change information accumulatingunit in the situation determining apparatus according to the firstembodiment of the invention.

FIG. 6 is a diagram illustrating the division of the image captured bythe camera CM according to the first embodiment of the invention intolocal areas.

FIG. 7 is a diagram illustrating an example of motion vectors extractedby a local image change detecting unit of the situation determiningapparatus according to the first embodiment of the invention.

FIG. 8 is a diagram illustrating elements of local change informationthat is detected by the local image change detecting unit and is thenaccumulated in the local image change information accumulating unit inthe situation determining apparatus according to the first embodiment ofthe invention.

FIG. 9 is a diagram illustrating elements of a local change ratio thatis calculated by a local image change ratio calculating unit and is thenaccumulated in a local image change ratio accumulating unit in thesituation determining apparatus according to the first embodiment of theinvention.

FIG. 10 is a diagram illustrating the relationship among the frame imageaccumulated in the image accumulating unit, the local change informationaccumulated in the local image change information accumulating unit, thelocal change ratio accumulated in the local image change ratioaccumulating unit, and time in the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 11 is a diagram illustrating an example of a local image changeratio histogram calculated by a local image change ratio histogramcalculating unit of the situation determining apparatus according to thefirst embodiment of the invention.

FIG. 12 is a diagram illustrating an example of an image including themovement situation of a large number of persons in the operation of thesituation determining apparatus according to the first embodiment of theinvention.

FIG. 13 is a diagram illustrating an example of an image including themovement situation of a small number of persons in the operation of thesituation determining apparatus according to the first embodiment of theinvention.

FIG. 14 is a diagram illustrating an example of an image including themovement situation of persons in a place where there is a line ofpersons for waiting a train, in the operation of the situationdetermining apparatus according to the first embodiment of theinvention.

FIG. 15 is a table illustrating the number of regions that are changedand a local image change ratio in three situations, in the operation ofthe situation determining apparatus according to the first embodiment ofthe invention.

FIG. 16 is a diagram illustrating the characteristics of a local imagechange ratio histogram when a large number of persons move, in theoperation of the situation determining apparatus according to the firstembodiment of the invention.

FIG. 17 is a diagram illustrating the characteristics of a local imagechange ratio histogram when a small number of persons move, in theoperation of the situation determining apparatus according to the firstembodiment of the invention.

FIG. 18 is a diagram illustrating the characteristics of a local imagechange ratio histogram when a moving route of persons is deflected toone side, in the operation of the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 19 is a diagram illustrating the division of an image including themovement situation of a large number of persons into local areas in theoperation of the situation determining apparatus according to the firstembodiment of the invention.

FIG. 20 is a diagram illustrating the division of an image including themovement situation of a small number of persons into local areas in theoperation of the situation determining apparatus according to the firstembodiment of the invention.

FIG. 21 is a diagram illustrating the division of an image including asituation in which a moving route of persons is deflected to one sideinto local areas, in the operation of the situation determiningapparatus according to the first embodiment of the invention.

FIG. 22 is a diagram illustrating an example of a local image changeratio histogram calculated from an actual moving image in the operationof the situation determining apparatus according to the first embodimentof the invention.

FIG. 23 is a block diagram illustrating the internal structure of asituation determining unit of the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 24 is a diagram illustrating another example of the process of thelocal image change detecting unit of the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 25 is a diagram illustrating another example of the process of thelocal image change detecting unit of the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 26 is a diagram illustrating another example of the process of thelocal image change detecting unit of the situation determining apparatusaccording to the first embodiment of the invention.

FIG. 27 is a block diagram illustrating the internal structure of asituation determining unit of a situation determining apparatusaccording to a second embodiment of the invention.

FIG. 28 is a diagram illustrating the value of an index calculated inthe extraction of the amount of features by a feature extracting unit ofthe situation determining apparatus according to the second embodimentof the invention.

FIG. 29 is a diagram illustrating the amount of features extracted bythe feature extracting unit of the situation determining apparatusaccording to the second embodiment of the invention.

FIG. 30 is a diagram illustrating the distribution of the amount oftwo-dimensional features extracted from three scenes of images by thefeature extracting unit of the situation determining apparatus accordingto the second embodiment of the invention.

FIG. 31 is a diagram illustrating the relationship between the amount offeatures and the distribution of the amount of two-dimensional featuresextracted by the feature extracting unit of the situation determiningapparatus according to the second embodiment of the invention.

FIG. 32 is a diagram illustrating six scenes of images captured at aplatform of a station which are used to describe a congestiondetermining process of the situation determining apparatus according tothe second embodiment of the invention.

FIG. 33 is a diagram illustrating the distribution of the amount offeatures corresponding to six scenes in a two-dimensional feature spacewhich is used to describe the congestion determining process of thesituation determining apparatus according to the second embodiment ofthe invention.

FIG. 34 is a diagram illustrating the distribution of the amount offeatures in three situations, which is used to describe the congestiondetermining process of the situation determining apparatus according tothe second embodiment of the invention, and subspaces of thedistribution obtained in the three situations.

FIG. 35 is a diagram illustrating a subspace method which is used todescribe the congestion determining process of the situation determiningapparatus according to the second embodiment of the invention.

FIG. 36 is a diagram illustrating the association between a position ina subspace and an index indicating the degree of congestion, which isused to describe the congestion determining process of the situationdetermining apparatus according to the second embodiment of theinvention.

FIG. 37 is a diagram illustrating the distribution of the amount offeatures in four situations, which is used to describe the congestiondetermining process of the situation determining apparatus according tothe second embodiment of the invention, and subspaces of thedistribution obtained in the four situations.

FIG. 38 is a block diagram schematically illustrating the structure ofan abnormality determining apparatus according to a third embodiment ofthe invention.

FIG. 39 is a flowchart illustrating an abnormality determining processof the abnormality determining apparatus according to the thirdembodiment of the invention.

FIG. 40 is a graph illustrating the degree of congestion output from asituation determining apparatus of the abnormality determining apparatusaccording to the third embodiment of the invention and a change in thekind of situations in a normal state over time.

FIG. 41 is a block diagram schematically illustrating the structure of acongestion estimating apparatus according to a fourth embodiment of theinvention.

FIG. 42 is a diagram illustrating a method of calculating a process areain the congestion estimating apparatus shown in FIG. 41.

FIG. 43 is a diagram illustrating an example of the division of aprocess area into movement process areas by the congestion estimatingapparatus shown in FIG. 41.

FIG. 44 is a diagram illustrating an example of the division of theprocess area into texture process areas by the congestion estimatingapparatus shown in FIG. 41.

FIG. 45 is a block diagram illustrating the detailed structures of amovement information generating unit, a reference movement informationgenerating unit, and a movement information determining unit of thecongestion estimating apparatus shown in FIG. 41.

FIG. 46 is a diagram illustrating an example of a reference motionvector map stored in a reference motion vector map reference unit of thereference movement information generating unit shown in FIG. 45.

FIG. 47 is a diagram illustrating an example of a reference differencearea map stored in a reference difference area map reference unit of thereference movement information generating unit shown in FIG. 45.

FIG. 48 is a diagram illustrating an input image for a textureinformation determining process of the congestion estimating apparatusshown in FIG. 41.

FIG. 49 is a diagram illustrating the result of a texture featureextracting process for converting the input image for the textureinformation determining process of the congestion estimating apparatusshown in FIG. 41.

FIG. 50 is a diagram illustrating the flow of a process of calculating areference texture feature amount in a scene in which there is a person,which is used for the texture information determining process of thecongestion estimating apparatus shown in FIG. 41.

FIG. 51 is a diagram illustrating the flow of the process of calculatingthe reference texture feature amount in a scene in which there is noperson, which is used for the texture information determining process ofthe congestion estimating apparatus shown in FIG. 41.

FIG. 52 is a diagram illustrating a similarity calculating process forthe texture information determining process of the congestion estimatingapparatus shown in FIG. 41.

FIG. 53 is a diagram illustrating a method of determining the state ofan area in the congestion estimating apparatus shown in FIG. 41.

FIG. 54 is a diagram illustrating an example of the determination resultof the state of an area by the congestion estimating apparatus shown inFIG. 41.

FIG. 55 is a diagram schematically illustrating the structure of acongestion estimating apparatus according to a fifth embodiment of theinvention.

FIG. 56 is a diagram illustrating an example of the installation of acamera connected to the congestion estimating apparatus shown in FIG.55.

FIG. 57 is a flowchart illustrating a process of acquiring the approach,stop, and departure timings of a train in the congestion estimatingapparatus shown in FIG. 55.

FIG. 58 is a diagram illustrating a group of station platform sceneswhen a vehicle approaches.

FIG. 59 is a diagram illustrating the process result of the motionvector of the station platform by the congestion estimating apparatusshown in FIG. 55 when the vehicle stops.

FIG. 60 is a diagram schematically illustrating the structure of acongestion estimating apparatus according to a sixth embodiment of theinvention.

FIG. 61 is a diagram illustrating an example of the determination resultof the state of an area by the congestion estimating apparatus shown inFIG. 60.

FIG. 62 is a time-series graph illustrating the congestion index of thecongestion estimating apparatus shown in FIG. 60.

FIG. 63 is a congestion index time-series graph illustrating the overlapbetween the approach, stop, and departure timings of a train in thecongestion estimating apparatus shown in FIG. 60.

FIG. 64 is a flowchart illustrating a process of determining thetendency of the movement of persons in the congestion estimatingapparatus shown in FIG. 60.

FIG. 65 is a diagram illustrating the process of determining thetendency of the movement of persons in the congestion estimatingapparatus shown in FIG. 60.

DESCRIPTION OF REFERENCE NUMERALS AND SIGNS

-   100: IMAGE INPUT UNIT-   110: IMAGE ACCUMULATING UNIT-   120: LOCAL IMAGE CHANGE DETECTING UNIT-   130: LOCAL IMAGE CHANGE INFORMATION ACCUMULATING UNIT-   140: LOCAL IMAGE CHANGE RATIO CALCULATING UNIT-   150: LOCAL IMAGE CHANGE RATIO ACCUMULATING UNIT-   160: LOCAL IMAGE CHANGE RATIO HISTOGRAM CALCULATING UNIT-   170: SITUATION DETERMINING UNIT-   200: REFERENCE HISTOGRAM STORAGE UNIT-   210: HISTOGRAM COMPARING UNIT-   300: FEATURE EXTRACTING UNIT-   310: IDENTIFICATION REFERENCE STORAGE UNIT-   320: IDENTIFYING UNIT-   500: SITUATION DETERMINING APPARATUS-   510: TRAIN ARRIVAL DETECTING UNIT-   520: ABNORMALITY DETERMINING UNIT-   530: NOTIFYING UNIT-   610A, 610B, 610C: CONGESTION ESTIMATING APPARATUS-   611: IMAGE GENERATING UNIT-   612: AREA DIVIDING UNIT-   613: MOVEMENT INFORMATION GENERATING UNIT-   614: REFERENCE MOVEMENT INFORMATION GENERATING UNIT-   615: TEXTURE INFORMATION GENERATING UNIT-   616: REFERENCE TEXTURE INFORMATION GENERATING UNIT-   617: STORAGE UNIT-   618: MOVEMENT INFORMATION DETERMINING UNIT-   619: TEXTURE INFORMATION DETERMINING UNIT-   620: STAYING DETERMINING UNIT-   621: TIMING GENERATING UNIT-   622: ABNORMALITY DETERMINING UNIT-   630: RECTANGULAR AREA 1-   631: RECTANGULAR AREA 2-   640: IMAGE BUFFER UNIT-   641: OPTICAL FLOW CALCULATING UNIT-   642: FLOW REPRESENTATIVE DIRECTION AND SIZE CALCULATING UNIT-   643: EDGE EXTRACTING UNIT-   644: INTER-EDGE-FRAME DIFFERENCE UNIT-   645: REFERENCE MOTION VECTOR MAP REFERENCE UNIT-   646: REFERENCE DIFFERENCE AREA MAP REFERENCE UNIT-   647: MOTION VECTOR STATE DETERMINING UNIT-   648: DIFFERENCE AREA STATE DETERMINING UNIT-   649: MOVEMENT AREA STATE DETERMINING UNIT-   650: REFERENCE MOTION VECTOR MAP-   660: REFERENCE DIFFERENCE AREA MAP-   670: INPUT IMAGE-   680: TEXTURE FEATURE EXTRACTING PROCESS RESULT-   690: SCENE 1 IN WHICH THERE IS PERSON 691, 701: TEXTURE PROCESS AREA-   700: SCENE 1 IN WHICH THERE IS NO PERSON-   706: SIMILARITY CALCULATION-   710: MOVEMENT INFORMATION DETERMINING UNIT DETERMINES THAT THERE IS    MOVEMENT-   711: MOVEMENT INFORMATION DETERMINING UNIT MOTION DETERMINES THAT    THERE IS NO MOVEMENT-   712: TEXTURE INFORMATION DETERMINING UNIT DETERMINES THAT THERE IS    PERSON-   713: TEXTURE INFORMATION DETERMINING UNIT DETERMINES THAT THERE IS    NO PERSON-   714, 720, 762: MOVEMENT AREA-   715, 723, 764: NOISE AREA-   716, 721, 763: STAYING AREA-   717, 722: BACKGROUND AREA-   730: CAMERA-   760: CONGESTION INDEX TIME-SERIES GRAPH-   795: REFERENCE STAYING AREA-   796: STAYING START-   PH: PLATFORM OF STATION-   WL: SIDE WALL OF PLATFORM-   ST: STAIRCASE THROUGH WHICH PERSONS MOVE TO PLATFORM-   RL: RAILROAD LINE-   CM: CAMERA-   SD: SITUATION DETERMINING APPARATUS-   AR1 TO AR3: ARROW INDICATING MOVING ROUTE OF PERSONS-   WP: PERSONS WAITING FOR TRAIN-   MP: MOVING PERSONS

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, exemplary embodiments of the invention will be described indetail with reference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram schematically illustrating the structure of asituation determining apparatus according to a first embodiment of theinvention. In FIG. 1, the situation determining apparatus according tothis embodiment includes an image input unit 100, an image accumulatingunit 110, a local image change detecting unit 120, a local image changeinformation accumulating unit 130, a local image change ratiocalculating unit 140, a local image change ratio accumulating unit 150,a local image change ratio histogram calculating unit 160, and asituation determining unit 170.

FIG. 2 is a diagram illustrating an example in which the situationdetermining apparatus shown in FIG. 1 is installed at a platform of arailroad station. The railroad station includes a platform PH, a sidewall WL of the platform, a staircase ST that is disposed at an entranceto the platform, and a railroad line RL. A camera CM that captures theimages of persons at the platform PH is provided such that the axialdirection is aligned with the longitudinal direction of the platform PH,and is connected to a situation determining apparatus SD. FIG. 3 showsan example of the image captured by the camera CM. The angle of view,position, and optical axis direction of the camera CM are determinedsuch that the captured images includes the platform PH, an entrance ofthe staircase ST, and the side wall WL. The situation determiningapparatus SD corresponds to the situation determining apparatus shown inFIG. 1.

Next, the operation of the situation determining apparatus according tothis embodiment will be described with reference to a flowchart shown inFIG. 4. First, an image input step S100 is performed by the image inputunit 100. In Step S100, one frame of image captured by the camera CM ischanged into a format that can be digitally processed, and the image isaccumulated in the image accumulating unit 110. When the camera CM is ananalog camera, an analog image is converted into a digital image, and acompression process, such as an encoding process, is performed on thedigital image, if necessary. The processed image is accumulated in theimage accumulating unit 110. When the camera CM is a digital camera, theimage is input through a digital line and is accumulated in the imageaccumulating unit 110. In this embodiment, it is assumed that a 10-fpsdigital moving image is input and frame images at the current time aresequentially accumulated. When the camera CM is an analog camera, theimage accumulating unit 110 may be, for example, a VTR that accumulatesanalog images, and AD conversion may be performed immediately before theanalog image is output to the local image change detecting unit 120.

Then, a local image change detecting step S110 is performed by the localimage change detecting unit 120. In this step, as shown in FIG. 5, afirst time interval TS₁ is set to 0.1 (second) which is equal to animaging period, two frame images, that is, an image at the current timet_(k) and an image t_(k−1) of a time TS₁ ago, among the frame imagesaccumulated in the image accumulating unit 110 are extracted, and achange in a local area is detected. A method of dividing local areas ispredetermined according to the installation of the camera CM, as shownin FIG. 6. In this embodiment, it is assumed that the total number oflocal areas is N_(R). In order to correct the influence of perspectiveprojection, the size of the local area is set such that it is large at apoint close to the camera (a lower side of a screen) and is small at apoint far from the camera (an upper side of the screen). A method ofdetecting a change using a motion vector will be described below.

(1) Calculation of Motion Vector of Each Pixel in Two Frame Images

For example, a gradient method, such as a Lucus-Kanade method disclosedin Non-Patent Document 1, or a block matching method may be used tocalculate the motion vector. In this embodiment, it is preferable that aGood Features to Track method disclosed in Non-Patent Document 2 be usedor only a motion vector with high reliability be used in the subsequentprocess by evaluation values (for example, SAD and SSD) during matching.FIG. 7 shows an example of a motion vector with high reliability amongthe motion vectors calculated for the entire screen from two frameimages.

-   Non-Patent Document 1: B. D. Lucas and T. Kanade. “An iterative    image registration technique with an application to stereo vision”,    IJCAI, 1981.-   Non-Patent Document 2: Jianbo Shi, Carlo Tomasi, “Good Features to    Track”, IEEE Conference on Computer Vision and Pattern Recognition,    pp. 593-600, 1994 (CVPR'94).

(2) Integration of Motion Vectors in Each Local Area

Hereinafter, a suffix “k” is added to the motion vector extracted fromtwo frame images, that is, an image at the current time t_(k) and animage t_(k−1) of the time TS₁ ago, a suffix “i” is added to the motionvector included in an i-th local area (1≦i≦N_(R)), and a suffix “j” isadded to a j-th motion vector (1≦j≦NV_(k,i)) among NV_(k,i) motionvectors included in the i-th local area, which are used to represent amotion vector (u_(k,i,j), v_(k,i,j)).

(Equation 1) and (Equation 2) are used to calculate the average value ofthe motion vectors, thereby calculating a representative motion vector(mu_(k,i), mv_(k,i)) of a local area i. The length of the motion vectoron the image is affected by perspective projection and is smaller thanthe actual movement speed of a person as it is close to the upper sideof the screen. Therefore, as in (Equation 3) and (Equation 4), after theinfluence of the perspective projection is corrected, the motion vectorsmay be averaged to calculate the representative motion vector. Here,w_(k,i,j) is a weight coefficient for correcting the size of a j-thmotion vector in an i-th local area at the time t_(k). The size of themotion vector is set such that it is increased as the start point of themotion vector is closer to the upper side of the screen.

$\begin{matrix}{{mu}_{k,i} = {\frac{1}{{NV}_{k,i}}{\sum\limits_{j = 1}^{{NV}_{k,i}}u_{k,i,j}}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack \\{{mv}_{k,i} = {\frac{1}{{NV}_{k,i}}{\sum\limits_{j = 1}^{{NV}_{k,i}}v_{k,i,j}}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack \\{{mu}_{k,i} = {\frac{1}{{NV}_{k,i}}{\sum\limits_{j = 1}^{{NV}_{k,i}}\left( {w_{k,i,j} \cdot u_{k,i,j}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack \\{{mv}_{k,i} = {\frac{1}{{NV}_{k,i}}{\sum\limits_{j = 1}^{{NV}_{k,i}}\left( {w_{k,i,j} \cdot v_{k,i,j}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

(3) Threshold Value Process is Performed on Representative Motion Vectorto Determine Whether there is Movement

The size of the representative motion vector (mu_(k,i), mv_(k,i)) of thei-th local area is compared with a predetermined threshold value todetermine whether there is a movement. When the size of therepresentative motion vector is equal to or more than the thresholdvalue, it is determined that there is a movement. When the size of therepresentative motion vector is less than the threshold value, it isdetermined that there is no movement. At that time, local changeinformation M_(k,i) representing whether there is a change (motion) inthe local area i with a binary value is obtained. When it is determinedthat there is a movement, M_(k,i)=1, and if not, M_(k,i)=0. FIG. 8 showsan image representing local area local change information. In FIG. 8, alocal area in which there is a change is hatched. The local changeinformation M_(k) is accumulated in the local image change informationaccumulating unit 130 as binary vector information corresponding to thenumber of local areas that indicates whether there is a change in eachlocal area as represented by (Equation 5), as shown in FIG. 5. In FIG.5, a frame image I_(k) and the local image change information M_(k) areaccumulated at the time t_(k), and a frame image I_(k−1) and local imagechange information M_(k−1) are accumulated at the time t_(k−1).

M_(k)=[M_(k,1), M_(k,2), . . . , M_(k,N) _(R) ]^(T)  [Equation 5]

Returning to FIG. 4, a local image change ratio calculating step S120 isperformed by the local image change ratio calculating unit 140. In thisstep, a second time interval TS₂ is set to 10 (seconds). The change(movement) ratio of each local area at a previous time TS₂ (second) iscalculated. Since change information is calculated at the first timeinterval TS₁, the change ratio is calculated using TS₂/TS₁ changeinformation items. When TS₁=0.1 (second) and TS₂=10 (seconds), 100change information items are used. In FIG. 9, in the i-th local area,local change information items having a value of 1 (that is, there is achange) are counted among from the local change information M_(k,i) atthe time t_(k) to local change information M_(k−TS2/TS1+1,i) at a timet_(k−TS2/TS1+1). The count value is referred to as C_(k,i). The countvalue is divided by a total number TS₂/TS₁ to calculate a local changeratio RT_(k,i). The local change ratio RT_(k,i) may have a value [0,1].The local change ratio RT_(k) is accumulated in the local imageaccumulating unit 150 as vector information indicating the change ratioof each local area which corresponds to the number of local areas asrepresented by (Equation 6), as shown in FIG. 10. In FIG. 10, the localimage change ratio RT_(k) is accumulated at the time t_(k) and a localimage change ratio RT_(k−1) is accumulated at the time t_(k−1).

RT_(k)=[RT_(k,1), RT_(k,2), . . . , RT_(k,N) _(R) ]^(T)  [Equation 6]

Then, a local image change ratio histogram calculating step S130 isperformed by the local image change ratio histogram calculating unit160. In this step, a histogram of N_(R) local change ratios RT_(k,i) atthe current time t_(k) is calculated. Since the local change ratioRT_(k,i) can have a value of [0, 1], the number of classes is 1/BW whenthe width of the class of the histogram is BW. In this case, the widthBW of the class is 0.1, and the number of classes is 10. FIG. 11 showsan example of the calculated local image change ratio histogram. Ahorizontal axis indicates the local image change ratio and a verticalaxis indicates frequency (the number of areas). A total sum offrequencies (the integration value of the histogram) is a total numberN_(R) of areas.

In this embodiment, the movement situations of persons at the platformof the station shown in FIG. 2 are considered. The following threepatterns are considered as the movement situations.

(1) Free Movement of Large Number of Persons

For example, there is a situation in which many passengers get off thetrain after the train arrives and move toward the staircase ST (FIG.12). A large number of persons move through a plurality of moving routesAR1 to AR3.

(2) Free Movement of Small Number of Persons

For example, there is a situation in which the passengers get off thetrain pass through the staircase ST and move through the platform wherethere is no passenger waiting for the next train (FIG. 13). Since asmall number of persons move through an empty platform, the movingroutes of the persons are changed. As a result, similar to when a largenumber of persons move, a small number of persons move through aplurality of moving routes AR1 to AR3.

(3) Deflection of Moving Route to One Side

For example, there is a situation in which passengers waiting for thetrain are lined up and new passengers who have moved to the platformthrough the staircase ST move to the front side of the screen of theplatform through an empty space of the platform (FIG. 14). The movingroute of moving persons MP is limited to the vicinity of an arrow AR1since there are persons WP waiting for the train.

FIG. 15 shows the number of areas where a change occurs (which has achange ratio equal to or more than a predetermined value) and the sizeof the local image change ratio for these three movement situationpatterns. FIGS. 16, 17, and 18 show typical examples of the local imagechange ratio histograms shown in FIG. 11 for these three movingpatterns, such as (1) Movement of large number of persons, (2) Movementof small number of persons, and (3) Deflection of moving route to oneside, respectively.

First, the situation in which a large number of persons move will bedescribed supplementarily using the image shown in FIG. 19 that isobtained by performing a local area dividing process on the image shownin FIG. 12. When a large number of persons move, many areas in the imageare changed (moved). Since the density of persons is high, movingpersons pass through a certain local area one by one. Therefore, thelocal image change ratio is increased in many local areas, as shown in afirst column of FIG. 15. Similarly, in the local image change ratiohistogram shown in FIG. 16, there are a large number of local areashaving middle to large local image change ratios.

The situation in which a small number of persons move freely will bedescribed supplementarily using the image shown in FIG. 20 that isobtained by performing the local area dividing process on the imageshown in FIG. 13. Similar to when a large number of persons move, evenwhen a small number of persons move, the persons pass through variousroutes on an empty platform. Therefore, there is a change (motion) inmany areas of the image. However, since the density of persons is low,the number of moving persons passing through a certain local area issmall. Therefore, the local image change ratio is low as shown in asecond column of FIG. 15. Similarly, in the local image change ratiohistogram shown in FIG. 17, the number of local areas having a middlelocal image change ratio is large, but the number of local areas havinga large local image change ratio is small.

The situation in which the moving route is deflected to one side will bedescribed supplementarily using the image shown in FIG. 21 that isobtained by performing the local area dividing process on the imageshown in FIG. 14. When there is a line of persons waiting for the train,the moving route of persons is limited to the vicinity of the arrow AR1.Since the persons waiting for the train move little at those positionsbut do not move largely, an area in which a change (motion) occurs issubstantially limited to the area in which there are the moving personsMP, and the number of areas in which a change occurs is small. When thenumber of persons moving to the platform PH through the staircase ST isequal to that when a small number of persons move (FIG. 12) and themoving route is limited, the number of persons passing through themoving route per local area is larger than that when a small number ofpersons move, as shown in a third column of FIG. 15. Similarly, in thelocal image change ratio histogram shown in FIG. 18, the number of localareas having middle and large local image change ratios is smaller thanthat when a large number of persons move as shown in FIG. 16. However,unlike when a small number of persons move as shown in FIG. 17, there isat least a local area having a large local image change ratio.

As described above, the local image change ratio histogram is calculatedin the local image change ratio histogram calculating step S130 (localimage change ratio histogram calculating unit 160).

FIG. 22 shows local image change ratio histograms extracted from threescenes of an actual moving image (the movement of a large number ofpersons, the movement of a small number of persons, and the deflectionof moving routes to one side). Each of the local image change ratiohistograms corresponds to the histogram of a local change ratio RT_(k′)at a time t_(k′) (in this case, the local change ratio is calculated atthe time t_(k′), but as described with reference to FIG. 10, the localchange ratio at a certain time is calculated from the previous TS₂/TS₁local change information items). FIG. 22 shows the results when thetotal number N_(R) of local areas=162, TS₁=0.1 (second),TS₂=10(seconds), and the width BW of the class of the histogram=0.1. Ascan be seen from FIG. 22, the tendency is the same as those shown inFIGS. 16, 17, and 18.

Then, a situation determining step S140 is performed by the situationdetermining unit 170. In three situations, such as the movement of alarge number of persons, the movement of a small number of persons, andthe deflection of moving routes to one side, the local image changeratio histograms have the respective shapes shown in FIGS. 16, 17, and18, as described above. In this embodiment, the local image change ratiohistograms are calculated in advance in the above-mentioned threesituations and are then stores as reference histograms, and thereference histograms are compared with a local image change ratiohistogram in the determined situation, thereby determining one of thethree situations.

FIG. 23 is a diagram illustrating the internal structure of thesituation determining unit 170 according to this embodiment. Thesituation determining unit 170 includes a reference histogram storageunit 200 and a histogram comparing unit 210. At least one referencelocal image change ratio histogram, which has been calculatedpreviously, corresponding to each situation is stored in the referencehistogram storage unit 200 so as to be associated with the correspondingsituation (any one of the movement of a large number of persons, themovement of a small number of persons, and the deflection of movingroutes to one side). The histogram comparing unit 210 compares the localimage change ratio histogram in the situation to be determined that iscalculated by the local image change ratio histogram calculating unit160 with the reference histograms stored in the reference histogramstorage unit 200. Then, it is determined which of the local image changeratio histograms in non-determined situations is most similar to thereference histogram, and the situation associated with the most similarhistogram is output as the determination result of the situation. Forexample, histogram intersection of (Equation 7), a Bhattaccharyyacoefficient of (Equation 8), or normalized correlation of (Equation 9)is used as a method of calculating similarity between the histograms.

$\begin{matrix}{s = {\sum\limits_{u = 1}^{m}{w_{u}{\min \left( {p_{u},q_{u}} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack \\{s = {\sum\limits_{u = 1}^{m}\sqrt{p_{u} \cdot q_{u}}}} & \left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack \\{s = \frac{\sum\limits_{u = 1}^{m}{p_{u} \cdot q_{u}}}{\sqrt{\sum\limits_{u = 1}^{m}{p_{u}\sqrt{\sum\limits_{u = 1}^{m}q_{u}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack\end{matrix}$

Returning to FIG. 4, finally, Step S150 is performed by a control unit(not shown in FIG. 1). When the operator of the apparatus uses an inputunit (not shown in FIG. 1) to input a process end instruction, theprocess returns to the image input step S100 to process the next frameimage. When the process end instruction is not input, the process ends.

In this embodiment, in the local image change detecting step S110performed by the local image change detecting unit 120, the motionvector is used in order to detect a change in the local area between twoframe images separated at a time interval TS₁ among the frame imagesaccumulated in the image accumulating unit 110. However, other methodsmay be used. When the motion vector is used, it is possible to find themoving direction or speed. As a result, it is possible to moreaccurately determine situations or the degree of congestion. Thedifference between frames may be used to simply detect whether there isa change. As shown in FIG. 24, when the difference between two framescaptured at the time t_(k−1) and the time t_(k) is calculated, an imageD_(k)(x, y) in which the value of each pixel is composed of a differencein brightness (multiple values) is obtained. It may be determinedwhether there is a change in each local area on the basis of whether theaverage value d_(k,i) of the difference in brightness in the local areai is equal to or more than a predetermined threshold value. In FIG. 24,S_(i) indicates a set of pixels forming an area i and NP_(i) indicatesthe number of elements thereof.

As shown in FIG. 25, after the difference image D_(k)(x, y) in which thevalue of each pixel is composed of a difference in brightness isobtained, binarization indicating whether there is a change may beperformed to obtain a binarized difference image BD_(k)(x, y), and itmay be determined whether there is a change in each local area on thebasis of the number bd1 _(k,i) of pixels that are changed in the localarea. When the local areas have different areas, it may be determinedwhether there is a change in each local area on the basis of bd2 _(k,i)obtained by dividing (normalizing) the number of pixels that are changedby the area of the local area.

When the difference between frames is obtained on the basis of thebrightness value, the image is affected by a variation in illumination.Therefore, as shown in FIG. 26, the difference between the imagesobtained by performing edge extraction and binarization on an inputimage (XOR operation on each pixel) may be calculated to obtain aninter-frame difference image ED_(k)(x, y) at the edge, and it may bedetermined whether there is a change in each local area on the basis ofthe number ed1 _(k,i) of pixels that are changed in the local area. Whenthe local areas have different areas, it may be determined whether thereis a change in each local area on the basis of ed2 _(k,i) obtained bydividing (normalizing) the number of pixels that are changed by the areaof the local area.

As described above, according to the situation determining apparatus ofthis embodiment, the time change ratio of the brightness value in alocal area of the captured images and the histogram of the time changeratios of a plurality of local areas are analyzed to determine themovement situation of persons. In particular, it is possible to simplydetect that the moving route is deflected to one side. Therefore, forexample, in the place where there is a line of persons waiting for thetrain, such as the station, it is possible to estimate the line ofpersons waiting for the train. When the deflection of the moving routeto one side is detected in a passage where persons are not generallylined up, it is possible to estimate an obstacle to the free movement ofpersons.

Second Embodiment

The structure of a situation determining apparatus according to a secondembodiment of the invention is the same as that of the situationdetermining apparatus according to the first embodiment of theinvention, which is shown in FIG. 1. Therefore, a description thereofwill be omitted. In the first embodiment, the situation determining unit170 has the internal structure shown in FIG. 23, but the situationdetermining apparatus according to this embodiment has the internalstructure shown in FIG. 27. That is, the situation determining unit 170includes a feature extracting unit 300, an identification referencestorage unit 310, and an identifying unit 320. The feature extractingunit 300 extracts the amount of features for determining a situation orthe degree of congestion from the histogram calculated by the localimage change ratio histogram calculating unit 160. The relationshipbetween the amount of features for determining a situation or the degreeof congestion, which is extracted by the feature extracting unit 300,and the kind of situation or a congestion index is stored in theidentification reference storage unit 310 in advance. The identifyingunit 320 identifies the kind of situation or the degree of congestion onthe basis of the amount of features extracted by the feature extractingunit 300 and the identification reference stored in the identificationreference storage unit 310.

The flowchart of a method of determining a situation according to thisembodiment is the same as that according to the first embodiment shownin FIG. 4. In this embodiment, the process up to the local image changeratio histogram calculating step S130 and the process of Step 150 arethe same as those in the first embodiment, and a description thereofwill be omitted.

Next, an operation in the situation determining step S140 will bedescribed with reference to a block diagram shown in FIG. 27. First, thefeature extracting unit 300 extracts the amount of features fordetermining a situation or the degree of congestion from the histogramcalculated by the local image change ratio histogram calculating unit160. In this case, the feature extracting unit 300 extracts the amountof two-dimensional features from the local image change ratio histogram.The amount of features is extracted by the following method.

(1) In the local image change ratio histogram, the number of areashaving a middle change ratio or more (threshold value TH₁) is counted,and the number of areas is RN₁.

(2) In the local image change ratio histogram, the number of areashaving a large change ratio (equal to or more than a threshold valueTH₂; TH₂>TH₁) is counted, and the number of areas is RN₂.

(3) If RN₁ is not equal to 0, (f₁, f₂)=(RN₁/N_(R), RN₂/RN₁) is theamount of features, and if RN₁ is equal to 0, (f₁, f₂)=(RN₁/N_(R), 0) isthe amount of features.

when TH₁=0.4 and TH₂=0.7, RN₁ and RN₂ in each of the three situationsshown in FIG. 22 (the movement of a large number of persons, themovement of a small number of persons, and the deflection of movingroutes to one side) have the values in the table shown in FIG. 28, andf₁ and f₂ in each of the three situations have the values in the tableshown in FIG. 29.

In addition, the relationship between the amount of features fordetermining a situation or the degree of congestion, which is extractedby the feature extracting unit 300, and the kind of situation or acongestion index is calculated in advance and is then stored in theidentification reference storage unit 310. FIG. 30 shows thedistribution of the amount of two-dimensional features (f₁, f₂) that isextracted in advance from three scenes of images. In the distribution,each point is extracted from the local change ratio histogram at acertain time. That is, features are extracted from each of the threescenes at a plurality of times. The distribution shown in FIG. 30 isused as information stored in the identification reference storage unit310.

The identifying unit 320 identifies the kind of situation or the degreeof congestion on the basis of the amount of features extracted from thefeature extracting unit 300 and the identification reference stored inthe identification reference storage unit 310.

Next, a method of identifying the kind of situation will be described.It is assumed that the information shown in FIG. 30 is stored in theidentification reference storage unit 310. It is assumed that each ofthe histograms calculated by the local image change ratio histogramcalculating unit 160 is any one of the histograms shown in FIG. 22 andthe kind of situation thereof is not known. In this case, the amount offeatures extracted by the feature extracting unit 300 is a set of valuesin any one of the rows in the table shown in FIG. 29. A feature point ofthe amount of features stored in the identification reference storageunit 310 that is closest to the amount of features extracted by thefeature extracting unit 300 is searched (nearest neighboring method),and the kind of situation associated with the feature point is output.In this way, it is possible to determine the situation of a scenecorresponding to the amount of features extracted by the featureextracting unit 300. FIG. 31 shows the relationship between each valueshown in the table of FIG. 29 and the distribution of the amount oftwo-dimensional features (f₁, f₂) shown in FIG. 30. It is possible toaccurately determine a situation using the nearest neighboring method.

The nearest neighboring method has been described above, but theinvention is not limited thereto. Any method may be used whichidentifies an unknown sample using a plurality of supervised trainingsamples. For example, a support vector machine (SVM), a discriminantanalysis method (linear discriminant analysis method or a secondarydiscriminant analysis method), or a neural network may be used. Sincethe methods are disclosed in various documents, a description thereofwill be omitted. For example, the SVM is disclosed in Non-PatentDocument 3 and the discriminant analysis method is disclosed inNon-Patent Document 4.

-   Non-Patent Document 3: Nello Cristianini, John Shawe-Taylor,    Translated by Tsuyoshi Ohkita, Kyoritsu Shuppan Co., Ltd., “An    introduction to Support Vector Machine”,-   Non-Patent Document 4: Haruo Yanai, and others, “Multivariate    Analysis, Modern Statistics 2”, Asakura Publishing Co., Ltd.,    January, 1979.

Next, a method of determining the degree of congestion will bedescribed. First, a change in indexes RN₁ and RN₂ and a change in theamount of features f₁ and f₂ during congestion will be described in thecase of free movement (when the moving route is not limited) and whenthe moving route is deflected to one side.

<In Case of Free Movement (when Moving Route is not Limited)>

When a state [1] is changed to a state [2] in FIG. 32 (in FIG. 32, anumerical number is in a circle, which is the same with other numericalvalues 3 to 6), the number RN₁ of areas having a middle local imagechange ratio or more is increased, but the number RN₂ of areas having alarge local image change ratio is not largely increased. The reason isthat, in the situation in which the moving route is limited, when thenumber of persons is increased, the persons pass through various routes(the persons are less likely to pass through the same route). Therefore,the position is changed from [1] to [2] in FIG. 33 in a two-dimensionalfeature amount space (f₁, f₂) due to an increase in the feature amountf₁.

In FIG. 32, a state [3] indicates that the moving routes of the personsare distributed substantially in the entire region of the platform ofthe station. In FIG. 32, when a state is changed from [2] to [3],similar to the change from [1] to [2], the position is changed from [2]to [3] in FIG. 33 in the two-dimensional feature amount space (f₁, f₂)due to an increase in the feature amount f₁.

In a state [4] of FIG. 32, since the density of persons is increased,the number of areas having a large local image change ratio isincreased. An increasing rate of the number RN₁ of areas having a middlelocal image change ratio or more is more than that of the number RN₂ ofareas having a large local image change ratio. In the two-dimensionalfeature amount space (f₁, f₂), the position is changed from [3] to [4]in FIG. 33 (the plotted points of the amount of features are more likelyto move).

When the state is changed in the order of [4]→>[3]→>[2]→>[1] in FIG. 32,the amount of features is moved in an opposite order of [4]→[3]→[2]→[1]in the feature amount space of FIG. 33.

<When Moving Route is Deflected to One Side>

The movement of the plotted points in the feature amount space of FIG.33 when the state is changed from [1] to [2] in FIG. 32 is the same asthat during the free movement. A line of persons waiting for the trainstarts to be formed from the state [2]. In the state [5] of FIG. 32 inwhich a line of persons starts to be formed, the moving route of thepersons is limited by the line of persons. Once the moving route islimited by a line of persons, the number RN₁ of areas having a middlelocal image change ratio or more reaches the peak. When the number ofpersons lined up is increased and the moving route is further limited,the index RN₁ is reduced. For the other index RN₂, when persons passesthrough an area at a constant rate, the number of persons passingthrough the area per unit time is increased since the moving route ofthe persons is limited, which results in an increase in the index RN₂.In the two-dimensional feature amount space (f₁, f₂) of FIG. 33, theposition is changed from [2] to [5].

In a state [6] of FIG. 32, the number of persons lined up is furtherincreased, and the moving route of the persons is further limited.Therefore, the number RN₁ of areas having a middle local image changeratio or more is reduced and the number RN₂ of areas having a largelocal image change ratio is increased. In the two-dimensional featureamount space (f₁, f₂) of FIG. 33, the position is changed from [5] to[6].

Next, a method of calculating the degree of congestion will bedescribed. Here, an example using a subspace method will be described.The details of the subspace method are disclosed in Chapter 14 ofNon-Patent Document 5. Here, only the outline of the subspace methodwill be described. First, processes (1) and (2) are performed.

(1) A plurality of feature amounts is extracted for each of the threesituations, that is, the movement of a large number of persons, themovement of a small number of persons, and the deflection of a movingroute to one side, thereby calculating the distribution shown in FIG.30.

(2) A main component analysis process is performed on thetwo-dimensional distribution of each situation, and a straight line of afirst main component (main axis) is used as a one-dimensional subspace.FIG. 34 shows the subspaces of three situations.

A method of determining a situation including the amount of featuresextracted from a scene from which the degree of congestion will becalculated, and a method of calculating the degree of congestion will bedescribed in (3).

(3) A subspace including the amount of features whose situation is notknown is determined. As shown in FIG. 35, when an input feature amountvector f is projected to a subspace, a subspace having the maximumprojection component ∥Pf∥ is selected as the identification result. Thiscorresponds to another embodiment that identifies the kind of situation.As shown in FIG. 36, a position on the subspace (straight line) and anindex indicating the degree of congestion are associated with each otherin advance, and the index indicating the degree of congestion can becalculated on the basis of the end position (the position of an arrowPOS in FIG. 35) of a vector when the feature amount vector f isprojected to the subspace. The closest congestion level (integer) inFIG. 36 may be selected as the index indicating the degree of congestionand interpolation may be performed at the end position to calculate adecimal congestion level.

-   Non-Patent Document 5: “Technical Review and View in Computer    Vision”, Takashi Matsuyama, Yoshinori Hisano, Atsushi Inomiya,    Shingijyutsu (New Technology) Communications, 1998/06

In this embodiment, the degree of congestion is calculated on the basisof the position of the feature vector projected to the subspace.However, instead of the main component analysis method, other methods,such as a multiple regression analysis method, may be used. The detailsof the multiple regression analysis are disclosed in Non-Patent Document4, and thus a description thereof will be omitted.

In this embodiment, three situations (the movement of a large number ofpersons, the movement of a small number of persons, and the deflectionof a moving route to one side) are defined as the kind of situations.However, two situations, that is, “normal movement” and “deflection of amoving route to one side” corresponding to the movement of a largenumber of persons and the movement of a small number of persons may beidentified. The situations may be classified into four or moresituations. For example, the situations may be classified into foursituations as shown in FIG. 37 such that an approximation error isreduced when approximation is performed in the subspace.

As described above, according to the situation determining apparatus ofthis embodiment, the time change ratio of the brightness value in alocal area of the captured images and the histogram of the time changeratios of a plurality of local areas are analyzed to determine themovement situation of persons and calculate the congestion level. Inparticular, it is possible to simply detect that the moving route isdeflected to one side. Therefore, for example, in the place where thereis a line of persons waiting for the train, such as the station, it ispossible to estimate the line of persons waiting for the train and thecongestion level thereof. In addition, in the situation in which themoving route is not deflected to one side, it is possible to estimatethe situation and the degree of congestion. When the deflection of themoving route to one side is detected in a passage where persons are notgenerally lined up, it is possible to estimate an obstacle to the freemovement of persons.

Third Embodiment

FIG. 38 is a block diagram schematically illustrating the structure ofan abnormality determining apparatus according to a third embodiment ofthe invention. In FIG. 38, the abnormality determining apparatusaccording to this embodiment includes a situation determining apparatus500, a train arrival detecting unit 510, an abnormality determining unit520, and a notifying unit 530. Similar to the first embodiment, theabnormality determining apparatus is installed at the platform of therailroad station shown in FIG. 2.

The situation determining apparatus 500 has the structure shown in theblock diagram of FIG. 1, and transmits the kind of situation determinedor the degree of congestion to the abnormality determining unit 520. Thedetailed operation of the situation determining apparatus 500 is thesame as that according to the first embodiment or the second embodiment.It is assumed that the kind of situation to be determined includes atleast a situation in which the moving route of persons is deflected toone side.

The train arrival detecting unit 510 detects the arrival of the train tothe platform. Any method may be used as long as it can detect thearrival of the train. For example, a method of recognizing an image, amethod using a sensor, and a method using train operation data may beused. In the method of recognizing the image, appearance information,such as the color and shape of the train, may be registered in advance,and template matching with the registered information may be performedto determine the arrival of the train. The method using the motionvector described in the first embodiment may also be used. When aplurality of motion vectors having similar direction and intensityappears in a predetermined area (in the vicinity of a railroad line) ofthe image, the movement of a rigid body may be detected to determine thearrival of the train. In the method using the sensor, for example, aload sensor is provided below the railroad and it is possible todetermine the arrival of the train on the basis of a load value. Inaddition, a laser light emitting device and a light receiving device maybe used to detect laser light reflected from the train or detect thatthe light emitted from the laser light emitting unit is shielded by thetrain, thereby determining the arrival of the train. As an example ofthe method using the train operation data, it is possible to detect thearrival of the train by receiving communication data from the train or atrain control center.

The abnormality determining unit 520 determines whether there isabnormality on the basis of both the determination result of thesituation and the determination result of the degree of congestion fromthe situation determining apparatus 500, and train arrival informationfrom the train arrival detecting unit 510. The details of this processwill be described below. When the abnormality determining unit 520determines that there is abnormality, the notifying unit 530 sends thefact to a predetermined contact address.

Next, the operation of the abnormality determining apparatus accordingto this embodiment will be described with reference to a flowchart shownin FIG. 39. An image input step S100 to a situation determining stepS140 are performed by the situation determining apparatus 500. Thestructure of the situation determining apparatus 500 is shown in FIG. 1,and units that perform Steps S100 to S140 and the operations thereof arethe same as those according to the first embodiment or the secondembodiment. Therefore, a detailed description thereof will be omitted.Here, a time-series variation in the congestion index and thedetermination result of the situation output from the situationdetermining apparatus 500 during a normal operation will be describedwith reference to FIG. 40. FIG. 40 is a diagram schematicallyillustrating a time-series variation in the degree of congestion and thedetermination result of the situation output from the situationdetermining apparatus according to the second embodiment.

<Congestion Index>

The horizontal axis indicates time, and the train arrives at a time t₃and a time t₆. Since the previous train has arrived at a time t₁, thereis no person at the platform and the degree of congestion is 0. Sincepersons move to the platform in order to get on the train up to a timet₃ where the train arrives, the degree of congestion is graduallyincreased. At the time t₃ where the train arrives, the passengerswaiting for the train get on the train, and the passengers get off fromthe train to the platform and move to the staircase ST shown in FIG. 2.At a time t₄ where all the passengers move to the staircase ST, thedegree of congestion returns to 0. Thereafter, the situation from thetime t₁ to the time t₄ is repeated.

<Determination Result of Situation>

During the period from the time t₁ to the time t₂, there is no line ofpersons waiting for the train. The situation is determined as “situation1: the movement of a small number of persons”. Since a line of personswaiting for the train starts to be formed from the time t₂, thesituation is determined as “situation 2: the deflection of a movingroute to one side”. When the train arrives at the time t₃, all thepassengers move to the staircase ST shown in FIG. 2. Therefore, thesituation is determined as “situation 3: the movement of a large numberof persons”.

Then, a train arrival detecting step S200 is performed by the trainarrival detecting unit 510. This process is the same as the operation ofthe train arrival detecting unit 510.

Then, Step S210 is performed by a control unit (not shown in FIG. 38).When there is train arrival information at the current time (in the caseof YES), the process proceeds to an abnormality determining step S220.When there is no train arrival information (in the case of NO), theprocess proceeds to Step S150.

Then, the abnormality determining step S220 is performed by theabnormality determining unit 520. This step is performed as follows. Thetime from the arrival of the train to the returning of the degree ofcongestion to 0 (for example, the period from the time t₃ to the timet₄) is measured in advance, and a threshold value DT is determined onthe basis of the measured value. When the congestion index is not lessthan a predetermined threshold value or the situation is not determinedas “situation 1: the movement of a small number of persons” after a timeDT has elapsed from the detection of the arrival of the train by thetrain arrival detecting unit 510, it is determined that there isabnormality.

Examples of the abnormality include when all the passengers waiting forthe train at the platform do not get on the arriving train and somepassengers are left at the platform since there is no spaces in thetrain and when congestion occurs in the staircase ST and the persons gotoff the train cannot move to the staircase ST even after the time DT haselapsed. The time until the congestion is removed is affected by thenumber of persons waiting for the train and the number of persons gotoff the train. Therefore, it is preferable to set the threshold value DTto an appropriate value according to the day and time.

Then, Step S230 is performed by the control unit (not shown in FIG. 38).If it is determined in the abnormality determining step that there isabnormality (in the case of YES), the process proceeds to a notifyingstep S240. If not (in the case of NO), the process proceeds to StepS150. The notifying step S240 is performed by the notifying unit 530.The notifying process is the same as the operation of the notifying unit530. Finally, Step S150 is performed by the control unit (not shown inFIG. 38). In FIG. 38, when the operator of the apparatus uses an inputunit (not shown in FIG. 38) to input a process end instruction (in thecase of YES), the process returns to the image input step S100 toprocess the next frame image. When the process end instruction is notinput (in the case of NO), the process ends.

As described above, according to the abnormality determining apparatusof this embodiment, it is possible to determine an abnormal congestionstate different from a normal congestion state on the basis of the kindof situation or the degree of congestion obtained by the situationdetermining apparatus.

Fourth Embodiment

FIG. 41 is a block diagram schematically illustrating the structure of acongestion estimating apparatus according to a fourth embodiment of theinvention. In FIG. 41, a congestion estimating apparatus 610A accordingto this embodiment includes: an image generating unit 611 that convertsan image of various scenes or an image captured by the camera into adigital image and outputs the digital image; an area dividing unit 612that divides an input image into partial areas; a movement informationgenerating unit 613 that generates movement information from an imagesequence of the image generating unit 611; a reference movementinformation generating unit 614 that stores and updates referencemovement information, which is a reference for a motion in each partialarea; a texture information generating unit 615 that generates textureinformation of the image output from the image generating unit 611; areference texture information generating unit 616 that stores andupdates reference texture information for determining whether there is aperson in each partial area; a storage unit 617 that stores thereference movement information or the reference texture information; amovement information determining unit 618 that compares the movementinformation output from the movement information generating unit 613with the reference movement information generated by the referencemovement information generating unit 614 to determine whether there is amovement in each partial area on the basis of the comparison result; atexture information determining unit 619 that compares the textureinformation output from the texture information generating unit 615 withthe reference texture information generated by the reference textureinformation generating unit 616 to determine whether there is the sametexture information as a person in each partial area; and a stayingdetermining unit 620 that receives the determination results from themovement information determining unit 618 and the texture informationdetermining unit 619 and determines whether there is a person in eacharea on the basis of the received determination results.

The operation of the congestion estimating apparatus 610A having theabove-mentioned structure will be described with reference to FIGS. 42,43, and 44. First, image data obtained by the image generating unit 611is transmitted to the movement information generating unit 613 and thetexture information generating unit 615. Here, the function of the imagedividing unit 612 will be described with reference to FIG. 42. First,the area dividing unit 612 performs area division for the movementinformation generating unit 613 and the reference movement informationgenerating unit 614. The area is determined on the basis of the size ofan object, which is a target. A rectangular area (1) 630 and arectangular area (2) 631 having object sizes corresponding to the frontand rear positions are set. A movement process area at an arbitraryposition is determined by linear interpolation on the basis of the settwo rectangular areas. FIG. 43 shows an example of the division of themovement process area.

Then, a texture process area for the texture information generating unit615 and the reference texture information generating unit 616 isdetermined on the basis of the division result of the area for thedetermined movement information. In the division of the texture processarea for texture, a set of a plurality of area division results formovement information is one texture process area. FIG. 44 shows anexample in which a total of four movement process area results formovement information, that is, two-by-two movement process area resultsbelong to one texture process area. In this embodiment, four movementprocess area results are used, but the number thereof may be freelychanged depending on the design.

Next, the operations of the movement information generating unit 613,the reference movement information generating unit 614, and the movementinformation determining unit 618 will be described with reference toFIG. 45. The image input from the image generating unit 611 is stored inan image buffer unit 640. Then, the image is input to an optical flowcalculating unit 641 and an edge extracting unit 643. The optical flowcalculating unit 641 extracts a feature point from the image andsearches a point on the next frame corresponding to the extractedfeature point. A vector connecting the corresponding feature point andthe feature point of the original image is a motion vector. A flowrepresentative direction/size calculating unit 642 calculates therepresentative flow and representative size of each movement processarea.

Each movement process area includes a plurality of motion vectors.However, the average of a plurality of motion vectors is calculated as arepresentative size, and when there is a movement vector having aplurality of directions, the most frequent directional vector iscalculated as a flow representative direction. When the direction iscalculated, a frequency distribution is created by only the flow havinga size equal to or more than a predetermined value to remove a flow ofnoise.

The representative direction information (unit: radian) of the flow andthe representative size (unit: pixel) of the flow for each movementprocess area are stored and are than transmitted to a motion vectorstate determining unit 647. The reference movement informationgenerating unit 614 includes a reference motion vector map referenceunit 645 and a reference difference area map reference unit 646. Areference motion vector map 650 shown in FIG. 46 is stored in thereference motion vector map reference unit 645, and each movementprocess area has a threshold value, which is a reference value forwhether there is a movement. For example, an area (1) 651, an area (2)652, an area (3) 653 (in FIG. 46, parentheses “( )” having a numericalvalue therein are omitted, which is the same with the followingdescription) have a threshold value of 10. When the flow has a size morethan the threshold value, it is determined that there is a movement.When the size of the flow is less than the threshold value, it isdetermined that there is no motion. Similarly, an area (8) 654 and anarea (9) 655 have a threshold value of 9. All the movement process areashave threshold values.

Returning to FIG. 45, the edge extracting unit 643 extracts the edge ofan input image using the existing edge extract program, such as a Sobelfilter, and transmits the extracted result to an inter-edge-framedifference unit 644. The inter-edge-frame difference unit 644 calculatesthe difference between the previous edge frame and the currently inputedge frame, extracts the pixels of the current frame that are differentfrom those of the previous frame, and transmits them to the differencearea state determining unit 648. The reference movement informationgenerating unit 614 includes the reference difference area map referenceunit 646, and the map includes a map for determining whether there is amovement when the pixels are moved in several percent of the movementprocess areas. FIG. 47 shows an example of a reference difference areamap 660.

In the reference difference area map 660, when 10% of the process area,such as an area (1) 661, an area (2) 662, or an area (3) 663 (in FIG.47, parentheses “( )” having a numerical value therein are omitted,which is the same with the following description), is moved, the area isdetermined to be a movement area. When the amount of movement is lessthan 10%, the area is determined to be a non-movement area. For example,an area (8) 664 and an area (9) 665 are determined to be movement areaswhen 15% of the area is moved. In this way, the difference area statedetermining unit 648 performs the above-mentioned determining process onall the movement process areas. The movement area state determining unit649 determines that there is a movement in the movement process areathat has been determined as a movement area on the basis of both thedetermination result of the motion vector state determining unit 647 andthe determination result of the difference area state determining unit648, and transmits the determination result to the staying determiningunit 620 (see FIG. 41). The movement process area that is determined tobe a non-movement area on the basis of both the determination result ofthe motion vector state determining unit 647 and the determinationresult of the difference area state determining unit 648 is transmittedas a non-movement area to the staying determining unit 620.

Next, the operations of the texture information generating unit 615, thereference texture information generating unit 616, and the textureinformation determining unit 619 will be described with reference toFIGS. 48 and 49. The texture information generating unit 615 extracts atexture feature amount from each of the texture process areas divided bythe feature area dividing unit 612. A frequency conversion process isperformed on the texture feature amount using Fourier transform. FIG. 48shows an example of an input image 670. The frequency conversion processis sequentially performed on an area (t1) 671, an area (t2) 672, area(t3) 673, . . . (in FIG. 48, parentheses “( )” having a numerical valuetherein are omitted, which is the same with the following description).FIG. 49 shows an example of the result of a texture feature amountextracting process. A texture feature amount extraction result 680 isobtained by performing the frequency conversion process on the area (t1)671, the area (t2) 672, the area (t3) 673, . . . as inputs (in FIG. 49,parentheses “( )” having a numerical value therein are omitted, which isthe same with the following description). The texture feature amountextraction result 680 includes an area t1 texture feature amount 681, anarea t2 texture feature amount 682, and an area t3 texture featureamount 683.

The reference texture information generating unit 616 extracts a texturefeature amount from each texture process area of a group of scenes inwhich there is a person and a group of scenes in which there is noperson. FIG. 50 shows a scene (1) 690 in which there is a person (inFIG. 50, parentheses “( )” having a numerical value therein are omitted,which is the same with the following description). A texture featureamount is extracted from each of the areas in the texture process area691, such as an area (t1) 692, . . . , an area (t17) 693, . . . . Inthis way, a texture feature amount 694 is extracted from the area t1 ofa scene 1 in which there is a person and a texture feature amount 695 isextracted from the area t17 of the scene 1 in which there is a person.Similarly, FIG. 51 shows a scene (1) 700 in which there is no person (inFIG. 51, parentheses “( )” having a numerical value therein are omitted,which is the same with the following description). A texture featureamount is extracted from each of the areas in the texture process area701, such as an area (t1) 702, . . . , an area (t17) 703, . . . . Inthis way, a texture feature amount 704 is extracted from the area t1 ofa scene 1 in which there is no person and a texture feature amount 705is extracted from the area t17 of the scene 1 in which there is noperson.

The process of the texture information determining unit 619 will bedescribed with reference to FIG. 52. The texture information determiningunit 619 outputs an area (t1) texture feature amount 681, . . . for anarea (t1) 671, . . . of each texture process area in the image 670 inputfrom the texture information generating unit 615 (in FIG. 52,parentheses “( )” having a numerical value therein are omitted, which isthe same with the following description). In addition, since thereference texture information generating unit 616 has a referencetexture feature amount corresponding to each texture area, for example,the texture feature amount 704 is extracted from the area t1 of thescene 1 in which there is no person and the texture feature amount 694is extracted from the area t1 of the scene 1 in which there is noperson. The texture feature amounts 704 and 694 correspond to the area(t1) 692 and the area (t1) 702 of the input image. Then, similaritycalculation 706 with the area t1 texture feature amount 681 isperformed. When similarity with the area of a scene in which there is noperson is high, the texture information determining unit 619 determinesthat there is no person in the area. When similarity with the area of ascene in which there is a person is high, the texture informationdetermining unit 619 determines that there is a person in the area. Inthe calculation of similarity, SAD (Sum of Absolute Difference) may beused to calculate texture with minimum similarity, or a normalizedcorrelation index may be used.

Next, the operation of the staying determining unit 620 will bedescribed with FIG. 53. In an area which is determined to be moved onthe basis of the determination result 710 of the movement informationdetermining unit, an area that is determined to have a person therein onthe basis of the determination result 712 of the texture informationdetermining unit is determined to be a movement area 714. In the areawhich is determined to be moved on the basis of the determination result710 of the movement information determining unit, an area that isdetermined to have no person therein on the basis of the determinationresult 713 of the texture information determining unit is determined tobe a noise area 715. In an area which is determined not to be moved onthe basis of the determination result 711 of the movement informationdetermining unit, the area that is determined to have a person thereinon the basis of the determination result 712 of the texture informationdetermining unit is determined to be a staying area 716. In the areawhich is determined not to be moved on the basis of the determinationresult 711 of the movement information determining unit, the area thatis determined to have no person therein on the basis of thedetermination result 713 of the texture information determining unit isdetermined to be a background area 717. In this way, it is possible tooutput the state of each area as shown in FIG. 54. FIG. 54 shows thestates of a movement area 720, a staying area 721, a background area723, and a noise area 724.

As described above, the movement information determining unit 618determines whether an area is moved and the texture informationdetermining unit 619 determines similarity using texture, therebydetermining an area in which there is a person or an area in which thereis no person. The staying determining unit 620 can discriminate variousstates of each area, a staying area, a movement area, a noise area, anda background area.

The texture information generating unit 615 and the reference textureinformation generating unit 616 of the congestion estimating apparatus610A according to this embodiment extract the texture feature amountusing a frequency conversion process by Fourier transform, but theinvention is not limited thereto. For example, edge information or Gaborfeatures may be used to extract the amount of features.

The reference texture information generating unit 616 of the congestionestimating apparatus 610A according to this embodiment generates a scenein which there is a person and a scene in which there is no person asone pattern of reference texture, but the invention is not limitedthereto. For example, a plurality of scenes in which there is a personand a plurality of scenes in which there is no person may be used togenerate a plurality of reference textures. Reference textures may begenerated from a plurality of scenes, such as a scene in which there area large number of persons, a scene in which there are a normal number ofpersons, a scene in which there are a small number of persons, and ascene in which there is no person, and the texture informationdetermining unit 619 may use them to determine an area in which there isa person or an area in which there is no person. In addition, aplurality of scenes may be learned by a learning method, such as asupport vector machine (SVM) or a boosting method, and therepresentative reference texture of a scene in which there is a personand the representative feature amount of a scene in which there is noperson may be used to determine an area in which there is a person or anarea in which there is no person.

Fifth Embodiment

FIG. 55 is a block diagram schematically illustrating the structure of acongestion estimating apparatus according to a fifth embodiment of theinvention. A congestion estimating apparatus 6108 according to thisembodiment differs from the congestion estimating apparatus 610Aaccording to the fourth embodiment in that it further includes a timinggenerating unit 621 that inputs the update timing of input referenceinformation to the reference movement information generating unit 614and the reference texture information generating unit 616. The otherstructures are the same as those according to the fourth embodiment, anda detailed description thereof will be omitted.

The process of the timing generating unit 621 gives the update timing tothe reference movement information generating unit 614 and the referencetexture information generating unit 616 at the approach, stop, anddeparture timings of the train will be described with reference to FIGS.56 and 57. As shown in FIG. 56, a camera 730 is installed at theplatform of the station such that it can monitor the railroad. When thetrain approaches the station on the railroad, a variation in therailroad during the approach of the train is detected using an imageprocessing technique, such as an inter-frame difference and backgrounddifference process, and the approach timing is transmitted to thereference movement information generating unit 614 and the referencetexture information generating unit 616. Immediately before the trainapproaches, there is a line of persons waiting for the train. Therefore,the reference texture information determining unit 619 uses a templatefor the existence of a person to update the reference texture. Then,when the train stops, the persons get on and off the train. Therefore,the reference movement information generating unit 614 uses the stoptiming of the train to update the reference movement information. FIG.57 shows the procedure of this process.

In a vehicle approach detecting step S141, the approach of a vehicle isdetected by image processing, and the detected result is transmitted toan update timing notifying step S145. In a vehicle stop detecting stepS142, after the approach of the vehicle is detected in the vehicleapproach detecting step S141, when there is no movement on the railroad,it is determined that the vehicle stops, and the vehicle stop time isnotified to the update timing notifying step S145. In a vehicledeparture detecting step S143, after the stop of the vehicle is detectedin the vehicle stop detecting step S142, when there is a movement on therailroad, it is determined that the vehicle departs, and the departureof the vehicle is notified to the update timing notifying step S145. Inthis process, the approach, stop, and departure of the vehicle arecontinuously detected until a process end instruction is issued (stepS144).

FIG. 58 shows an example of the image of the platform of the station atthe vehicle approach timing. As shown in FIG. 58, a plurality of scenesobtain at the approach timing is transmitted to the reference textureinformation generating unit 616, and a reference texture is generatedfrom the scene group. In this way, it is possible to adapt the referencetexture when there is a person to a scene. In addition, a referencetexture for each day and time is stored. In this way, it is possible todetermine texture information in synchronization with, for example, dayand time.

FIG. 59 shows an example of a sequence of the acquisition results ofmotion vectors after a process of extracting motion vectors from theplatform of the station at the vehicle stop timing. As shown in FIG. 59,a plurality of scenes obtained at the vehicle stop timing is transmittedto the reference movement information generating unit 614, and themagnitude of the movement of each movement process area or a directionalvector is extracted from the scene group at that time. Then, themovement at that time is used as a reference movement amount to acquirethe direction vector of each movement process area. In addition, thereference movement information for each day and time is stored. In thisway, it is possible to determine movement information in synchronizationwith, for example, day and time.

As described above, the timing generating unit 621 automatically setsthe reference movement amount of the movement information or the textureinformation in which there is a person on the basis of the approach,stop, and departure timings of the vehicle. In this way, it is possibleto discriminate a state in which there is a movement from a state inwhich there is no movement. In addition, it is possible to determinewhether there is a person or not.

The timing generating unit 621 of the congestion estimating apparatus6108 according to this embodiment generates the approach timing of thevehicle using an image processing technique, but the invention is notlimited thereto. For example, a detection sensor may be provided in therailroad, and information about the approach, stop, and departure of thevehicle may be obtained from the information of the sensor. In addition,the approach timing of the train may be notified in synchronization withthe arrival time in the train timetable.

The timing generating unit 621 uses the approach, stop, and departuretimings of the vehicle, but the invention is not limited thereto. Thetiming generating unit 621 may generate update timing at an arbitrarytime interval.

Sixth Embodiment

FIG. 60 is a block diagram schematically illustrating the structure of acongestion estimating apparatus according to a sixth embodiment of theinvention. A congestion estimating apparatus 610C according to thisembodiment differs from the congestion estimating apparatus 610Aaccording to the fourth embodiment in that it further includes a timinggenerating unit 621 that inputs the update timing of input referenceinformation to the reference movement information generating unit 614and the reference texture information generating unit 616 and anabnormality determining unit 622 that analyzes the time-series ratio ofa process area and determines whether there is abnormal congestion. Theother structures are the same as those of the congestion estimatingapparatus 610A according to the fourth embodiment, and a detaileddescription thereof will be omitted.

FIG. 61 shows a process area state determination result sequence 150output from the staying determining unit 620. The area statedetermination result sequence 750 has area determination results astime-series data of t(0)751, t(1)752, t(2)753, . . . , t(n)756 (in FIG.61, parentheses “( )” having a numerical value therein are omitted,which is the same with the following description). FIG. 62 shows acongestion index time-series graph 760 obtained by plotting the areastate determination result sequence 750. The graph is obtained byplotting a staying area 763, a movement area 762, a noise area 764, atotal congestion degree 761 in time series. The total congestion degree761 indicates the sum of the number of movement areas 762 and the numberof staying areas 763. The abnormal congestion determining unit 622determines that abnormal congestion occurs when the total congestiondegree 761 is more than a predetermined threshold value. For example,when the threshold value is set such that, if the total congestiondegree 761 is more than 80%, it is determined that the degree ofcongestion is abnormal, it is determined at the timing of t(5)754 andt(30)755 that abnormal congestion occurs, and the fact is notified. Inthis embodiment, abnormal congestion is determined on the basis of theratio of the total congestion degree 761. However, the abnormalcongestion may be determined on the basis of the ratio of the stayingarea 763. In this embodiment, abnormal congestion is determined on thebasis of the ratio of the total congestion degree 761. However, theabnormal congestion may be determined on the basis of the ratio of themovement area 762.

FIG. 63 shows a congestion index time-series graph obtained byoverlapping the timings of t10: vehicle approach 765, t20: vehicleapproach 768, t11: vehicle stop 766, t21: vehicle stop 769, t12: vehicledeparture 767, and t22: vehicle departure 770 obtained from the timinggenerating unit 621 on the congestion index time-series graph. Theabnormality determining unit 622 determines that there is abnormalitywhen the total congestion degree 761 is not reduced by a predeterminedthreshold value after a predetermined amount of time has elapsed fromthe timing of t10: vehicle approach 765 and the timing of t20: vehicleapproach 768 and notifies the fact. For example, in the range of apredetermined determination period 771 from the timing of t10: vehicleapproach 765, when the total congestion degree 761 is not reduced by 60%or more, the abnormality determining unit 622 determines that abnormalcongestion occurs, and notifies the fact.

In the example shown in FIG. 63, after the determination period 771 fromthe timing of t10: vehicle approach 765 and the timing of t20: vehicleapproach 768, the total congestion degree 761 is reduced by 60% or less,it is not determined that abnormal congestion occurs. In thisembodiment, abnormal congestion is determined on the basis of the ratioof the total congestion degree 761 at the vehicle approach timing.However, the abnormal congestion may be determined on the basis of theratio of the staying area 763. In this embodiment, abnormal congestionis determined on the basis of the ratio of the total congestion degree761. However, the abnormal congestion may be determined on the basis ofthe ratio of the movement area 762.

Next, a process of determining the tendency of the movement of persons,such as a staying start state, a staying removal state, and a normalstate, from the state of each process area determined by the stayingdetermining unit 620 will be described with reference to FIG. 64. In astaying area detecting step S180, a staying area is detected. In asearch step S181 of searching a neighboring area of the staying area, itis determined whether a new staying area is generated in a neighboringarea including a staying area or the staying area is removed, andinformation indicating the determination result is transmitted to astaying area continuous increase determining step S182 and a stayingarea continuous decrease determining step S185.

In the staying area continuous increase determining step S182, it isdetermined whether the staying area is continuously monotonouslyincreased and it is determined whether the number of staying areascontinuously increased is equal to or more than a threshold value (stepS183). If the number of staying areas is equal to more than apredetermined threshold value, in a staying start determinationnotifying step S184, a warning is issued before abnormal congestionoccurs due to an increase in the staying area. Similarly, in the stayingarea continuous decrease determining step S185, it is determined whetherthe staying area is continuously monotonously decreased and it isdetermined whether the number of staying areas continuously decreased isequal to or more than a threshold value (step S186). If the number ofstaying areas is equal to more than a predetermined threshold value, ina staying removal determination notifying step S187, it is notified thatthe congestion situation has been removed.

FIG. 65 shows an example of the above-mentioned process. At a time (t−4)790 and a time (t−3) 791 (in FIG. 65, parentheses “( )” having anumerical value therein are omitted, which is the same with thefollowing description), only the movement area is detected. At a time(t−2) 792, a staying area is generated, and the staying area isrecognized as a reference staying area 795. At a time (t−1) 793, a newstaying area is generated in the vicinity of the reference staying area795, a group of the staying area groups is used as a reference stayingarea group. At a time (t) 794, three new staying areas are generated inthe reference staying area group. When a threshold value for determininga staying start is three staying areas, it is determined at that timethat a staying start 796 occurs, and the fact is notified. The sameprocess as described above is performed on staying removal. Cases otherthan the staying start determination and the staying removaldetermination are determined to be a normal state. Therefore, the normalstate can be notified.

As described above, it is possible to determine the indexes ofcongestion situations (a staying start state, a staying removal state,and a normal state) and an abnormal state on the basis of various statesof each area, a staying area, a movement area, a noise area, and abackground area.

Although the exemplary embodiments of the invention have been describedabove, the invention is not limited thereto, but changes andapplications implemented by those skilled in the art on the basis of thespecification and known techniques are also included in the scope of theinvention.

The application is based on Japanese patent application No. 2007-278625filed on Oct. 26, 2007 and Japanese patent application No. 2007-280205filed on Oct. 29, 2007, the contents of which are incorporated hereintoby reference.

INDUSTRIAL APPLICABILITY

The situation determining apparatus, the situation determining method,and the situation determining program according to the invention canprovide an apparatus, a method, and a program capable of determining thekind of situation related to the movement of persons or the degree ofcongestion and achieving laborsaving and high efficiency when imagemonitoring is performed in a public space in which a plurality ofpersons is moved, such as a station or an airport. In addition, theabnormality determining apparatus, the abnormality determining method,and the abnormality determining program according to the invention canprovide an apparatus, a method, and a program capable of determining anabnormal congestion situation related to the movement of persons,achieving laborsaving and high efficiency when image monitoring isperformed in a public space in which a plurality of persons is moved,such as a station or an airport, and detecting abnormality in advance inorder to prevent accidents caused by abnormal congestion.

The congestion estimating apparatus according to the inventionautomatically sets the reference movement amount of movement informationto discriminate a state in which there is a movement and a state inwhich there is no movement. In addition, the congestion estimatingapparatus determines whether there is a movement and uses texture todetermine similarity, thereby discriminating various states of eacharea, a staying area, a movement area, a noise area, and a backgroundarea. Further, the congestion estimating apparatus uses the state ofeach area to estimate the degree of congestion, and can provide theindexes of congestion situations (a staying area, a movement area, anormal area, a staying start state, a staying removal state, and anormal state) and information about an abnormal state. Therefore, thecongestion estimating apparatus according to the invention can beapplied to an apparatus that notifies an abnormal congestion state.

1. A congestion estimating apparatus comprising: an area dividing unitthat divides a moving image into partial areas; a movement informationdetermining unit that determines whether or not there is a movement ineach of the partial areas; a person information determining unit thatdetermines whether or not there is a person in each of the partialareas; and a staying determining unit that receives determinationresults from the movement information determining unit and the personinformation determining unit to determine a state of area for each ofthe partial areas, wherein the staying determining unit determines thestate of area as a movement area in which there is a movement of personwhen the movement information determining unit determines that there isa movement and the person information determining unit determines thatthere is a person, the staying determining unit determines the state ofarea as a noise area when the movement information determining unitdetermines that there is a movement and the person informationdetermining unit determines that there is no person, the stayingdetermining unit determines the state of area as a staying area in whichthere is a person who is staying when the movement informationdetermining unit determines that there is no movement and the personinformation determining unit determines that there is a person, and thestaying determining unit determines the state of area as a backgroundarea in which there is no person when the movement informationdetermining unit determines that there is no movement and the personinformation determining unit determines that there no person.
 2. Thecongestion estimating apparatus according to claim 1, furthercomprising: a timing generating unit that receives the moving image anddetermines whether there is a person based on movement information,wherein, only when it is determined that there is a person, the timinggenerating unit gives update timing to the movement informationdetermining unit and to the person information determining unit.
 3. Thecongestion estimating apparatus according to claim 2, wherein the timinggenerating unit detects approach timing of a vehicle, and gives theupdate timing to the movement information determining unit and to theperson information determining unit at each approach timing.
 4. Thecongestion estimating apparatus according to claim 2, wherein themovement information determining unit samples reference movementinformation at the timing notified by the timing generating unit to seta threshold value for reference movement information, the movementinformation determining unit determines that there is a movement whenthe reference movement information is more than the threshold value;whereas the movement information determining unit determines that thereis no movement when the reference movement information is not more thanthe threshold value.
 5. The congestion estimating apparatus according toclaim 1, wherein the person information determining unit performs afrequency conversion process on input information to determinesimilarity in a frequency domain.
 6. The congestion estimating apparatusaccording to claim 2, wherein the person information determining unitsamples reference person information at the timing notified by thetiming generating unit to set the reference person information, and theperson information determining unit determines similarity between personinformation generated from the moving image and the reference personinformation, and when it is determined that the person information issimilar to the reference person information, the texture informationdetermining unit determines that there is a person.
 7. The congestionestimating apparatus according to claim 1, wherein the stayingdetermining unit outputs information indicating a state of any one ofthe staying area, the movement area, the noise area, and the backgroundarea as the state of each partial area.
 8. The congestion estimatingapparatus according to claim 7, further comprising: an abnormalitydetermining unit that receives the information output from the stayingdetermining unit, and analyzes each state of area to determine whetherabnormal congestion occurs.
 9. The congestion estimating apparatusaccording to claim 8, wherein the abnormality determining unit countsvarious states of each area, the staying area, the movement area, thenoise area, and the background area output from the staying determiningunit, and when a congestion index, which is a sum of the number ofstaying areas and the number of movement areas, is reduced by apredetermined threshold value or more after the approach timing of thevehicle obtained by the timing generating unit, the abnormalitydetermining unit determines that abnormality occurs.
 10. The congestionestimating apparatus according to claim 8, wherein the abnormalitydetermining unit counts various states of each area, the staying area,the movement area, the noise area, and the background area output fromthe staying determining unit, and when a ratio of the staying area ismore than a predetermined value, the abnormality determining unitdetermines that abnormality occurs.
 11. The congestion estimatingapparatus according to claim 8, wherein the abnormality determining unitcounts various states of each area, the staying area, the movement area,the noise area, and the background area output from the stayingdetermining unit, and the abnormality determining unit determines atendency of the movement of person that indicates a staying start,staying removal, or a normal state, from ratios of the staying area andthe movement area in time series.
 12. A method in a congestionestimating apparatus, comprising: dividing a moving image into partialareas; determining whether or not there is a movement in each of thepartial areas; determining whether or not there is a person in each ofthe partial areas; and determining a state of area for each of thepartial areas based on results of determining whether or not there is amovement and determining whether or not there is a person, wherein thestate of area is determined as a movement area in which there is amovement of person when it is determined that there is a movement and itis determined that there is a person, the state of area is determined asa noise area when it is determined that there is a movement and it isdetermined that there is no person, the state of area is determined as astaying area in which there is a person who is staying when it isdetermined that there is no movement and it is determined that there isa person, and the state of area is determined as a background area inwhich there is no person when it is determined that there is no movementand it is determined that there no person.